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The  broad  goal  of  this  project  is  to  develop  new  computational  processes  for  leveraging 
visual  cues  for  image-based  scene  understanding,  and  we  seek  to  achieve  this  goal  through 
a  careful  analysis  of  reflectance,  light  transport,  and  imaging.  During  the  past  year,  we 
have  continued  our  exploration  of  micro  computer  vision  systems,  meaning  those  that  can 
accomplish  useful  visual  tasks  within  the  power  and  size  constraints  of  micro-scale  platforms 
like  micro  air  vehicles  and  micro  sensor  nodes.  We  have  also  developed  a  passive  method  for 
simultaneous  depth  and  color  imaging  that  exploits  a  new  statistical  model  of  joint  spatial 
and  spectral  structure  in  natural  images. 


1  Micro  Sensors 

Building  on  our  preliminary  results  from  Year  Two  of  this  award  [3,  5],  we  continued  our 
investigation  of  visual  sensors  for  platforms  with  severe,  micro- Watt  power  budgets  that  com¬ 
bine  optical  processing  and  complementary  post-capture  computation.  As  shown  in  Fig  1, 
these  sensors  are  comprised  of  an  array  of  optical  elements,  with  each  element  combining  a 
discriminative  optical  filter  and  refractive  optics  to  achieve  low-power  target  detection  over 
a  wide  portion  of  the  hemisphere  of  light  that  is  visible  to  a  planar  sensor  array. 


Figure  1:  A  miniaturized  class  of  wide-angle  sensors  developed  during  Year  Two  of  this 
award.  Arrays  of  these  sensors  handle  specific  detection  tasks.  A  refractive  slab  creates  a 
180°  held-of-view  due  to  Snell’s  law.  Attenuating  templates  in  the  viewing  path  allow  optical 
filtering  and  enable  vision  tasks  such  as  locating  edges,  tracking  targets  and  detecting  faces. 
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During  the  past  year,  we  collaborated  with  Prof.  Ken  Crozier  at  Harvard  University  and 
Dr.  Geoffrey  Barrows  at  Centeye  Inc.  to  design  and  fabricate  micro-scale  sensors  that  com¬ 
bine  micron-resolution  optical  templates  with  a  refractive  slab  to  enable  low-power  target 
detection  (Fig.  2).  We  also  demonstrated  real-time  autonomous  fiducial  target  detection  on 
a  small  autonomous  helicopter  (Fig.  3).  These  results  are  reported  in  a  journal  paper  that 
has  been  accepted  to  the  IEEE  Transactions  of  Pattern  Analysis  and  Machine  Intelligence  [4] . 
They  also  appear  online  (including  videos) :  http: / / www.koppal.com / microvisionsensors.html 
The  journal  paper  also  presents  refined  mathematical  analysis  of  the  sensor  design  space, 
including  a  closed-form  solution  for  optimal  sensor  dimensions,  and  a  new  spherical  design 
that  allows  spherical  optical  filtering  without  distortion. 
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Figure  2:  A  prototype  system  for  optical  filtering  with  micro-scale  templates  embedded 
in  a  refractive  (PDMS)  slab,  all  fabricated  with  lithographic  techniques.  Left:  Array  of  six 
optical  elements,  each  with  a  distinct  template  and  isolated  with  black  baffles,  shown  in  a 
glass  container.  Middle  and  right :  Close-ups  of  the  top  and  side  of  the  six-element  array 
taken  under  a  microscope.  Details  of  the  fabrication  process  in  are  in  [4], 


2  Passive  depth  sensing  from  a  spatio-spectral  model 

In  Years  One  and  Two  of  this  award,  we  devoted  significant  attention  to  the  task  of  charac¬ 
terizing  the  statistical  structure  that  exists  jointly  in  the  spatial  and  spectral  dimensions  of 
a  natural  image.  During  the  current  reporting  period,  we  applied  insights  garnered  from  this 
analysis  to  create  a  new,  passive,  image-based  depth  sensor  (he.,  a  type  of  passive  RGB-D 
camera) . 

As  shown  inset  in  the  top-left  of  Fig.  4,  we  propose  modifying  the  aperture  of  a  conven¬ 
tional  color  camera  so  that  the  effective  aperture  size  for  one  color  channel  is  smaller  than 
that  for  the  other  two.  This  produces  an  image  where  different  color  channels  have  different 
depths-of-field,  and  from  this  we  can  computationally  recover  scene  depth,  reconstruct  an 
all-focus  image  and  achieve  synthetic  re- focusing,  all  from  a  single  shot.  These  capabilities 
are  enabled  by  a  spatio-spectral  image  model  that  encodes  the  statistical  relationship  be¬ 
tween  gradient  profiles  across  color  channels.  This  approach  substantially  improves  depth 
accuracy  over  alternative  single-shot  coded-  aperture  designs,  and  since  it  avoids  introducing 
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Figure  3:  An  autonomous  helicopter  from  Centeye  Inc.,  equipped  with  our  op¬ 

tics  (Fig.  2)  glued  to  a  CMOS  photosensor  array.  With  all  sensing,  processing,  and 
actuation  being  computed  on-board,  the  system  can  recognize  and  respond  to  sim¬ 
ple  visual  patterns,  such  as  a  T-shaped  fiducial  target.  A  full  video  is  available  at 
http:  /  /  www.koppal.com  /  microvisionsensors.html. 


additional  spatial  distortions  and  is  light  efficient,  it  allows  high-quality  deblurring  and  lower 
exposure  times. 

We  demonstrated  these  benefits  with  comparisons  on  synthetic  data,  as  well  as  results 
on  images  captured  with  a  prototype  lens,  as  shown  in  Fig.  4.  The  results  were  reported  in 
a  technical  report  and  at  the  European  Conference  of  Computer  Vision  [1,  2],  and  source 
code  is  available  at  http://vision.seas.harvard.edu/ccap/. 
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Figure  4:  The  proposed  color-coded  aperture  generates  reliable  estimates  of  scene  depth 
with  greater  accuracy  than  color- neutral  approaches  (no  user  interaction  is  required),  while 
simultaneously  allowing  high-quality  deblurring  to  obtain  an  all-focus  color  image  that  is 
aligned  with  the  depth  map. 
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